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As  investigation  was  made  of  the  practicality  of  assessing  anti-submarine  warfare 
(ASW)  team  performance  by  means  of  measures  of  the  volume  of  communications.  A 
system  for  classifying  communications  was  developed  based  on  an  analysis  of  published 
data  on  communication  rates  (e.g.,  number  of  evaluative  messages  sent  per  minute)  of 
ASW  helicopter  crews.  Next,  communications  were  recorded  for  ship's  teams  during  two 
exercises  in  the  14A2  ASW  team  trainer.  Communication  rates  were  computed  for 
various  types  of  messages  over  the  ship-to-ship  and  ship-to-air  circuits.  Rates  were 
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compared  against  instructor  grades  for  individuals,  subteams,  and  teams.  Communication 
rates  on  the  intership  circuit  tended  to  be  negatively  correlated  with  grades,  primarily 
because  instructors  gave  lower  grades  to  teams  doing  excessive  talking.  Rates  on  the  ship 
to  air  circuit  were  positively  correlated  with  performance  on  the  later  exercise  where  two 
aircraft  were  used  rather  than  one  and  where  a  much  greater  volume  of  information 
needed  to  be  transmitted.  On  the  internal  circuits,  few  significant  relationships  were 
found  between  communication  rates  and  performance.  Implications  of  the  findings  for 
development  of  an  objective  performance  measurement  system  for  team  training  are 
discussed. 


UNCLASSIFIED 

MeumTV  CLMMrtCATtOM  or  THIS  0X.  (MWMD 


FOREWORD 


This  research  and  development  was  done  under  task  area  ZF63-522-001-010  (Devel¬ 
opments  in  Technology  for  Applications  in  Training  and  Education),  work  unit  ZF63-522- 
001-010-03.02  (Team  Training  for  Tactical  Environments).  A  major  goal  of  the  work  unit 
is  to  develop  an  objective  performance  assessment  system  for  antisubmarine  warfare 
team  training  exercises. 

This  effort  investigated  the  feasibility  of  using  objective,  quantitative  measures  of 
team  communications  (e.g.,  amount  of  information  sent  per  minute  on  the  61 JS  sonar 
circuit)  for  assessing  team  performance.  There  are  several  indications  from  prior 
research  that  such  communication  rates  are  related  to  team  effectiveness.  The  project 
demonstrated  that  such  relationships  exist  for  surface  ship  ASW  but  are  not  strong  enough 
for  communication  rates  to  be  used  by  themselves  as  effective  measures  of  team 
performance.  The  analysis  also  indicates  that  a  single  measure  of  combat  team 
performance,  such  as  weapon  accuracy,  is  of  limited  importance  in  team  assessment 
because  of  the  complexity  of  ASW  team  exercises.  Composite  measures  are  needed  that 
reflect  the  many  significant  responses  that  a  team  must  make  to  perform  effectively. 

Data  from  this  effort  were  used  to  assist  in  development  of  ASW  team  training 
objectives  (NPRDC  Tech.  Note  81-18  and  Spec.  Rep.  82-4). 

Results  of  this  work  are  intended  primarily  for  the  Fleet  Anti-Submarine  Warfare 
Training  Center,  Pacific,  and  for  other  agencies  concerned  with  surface  ship  ASW 
training.  The  results  should  also  be  of  interest  to  the  Chief  of  Naval  Education  and 
Training,  the  Chief  of  Naval  Operations  (OP -01),  and  the  research  and  development 
community  concerned  with  team  training. 

Appreciation  is  expressed  to  personnel  of  the  Fleet  Anti-Submarine  Warfare  Training 
Facility,  San  Diego,  who  were  instrumental  in  facilitating  the  gathering  of  data  and 
providing  advice  on  technical  matters.  Particular  appreciation  is  extended  to  CAPT  R.  F. 
Comer,  for  his  support;  to  OSCM  Lowell  Johnson,  STGCS  Wayne  Gangstad,  STGC  Gerald 
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SUMMARY 


Problem 


Current  methods  of  assessing  the  performance  of  individuals  and  teams  during  team 
training  are  highly  subjective.  Objective  methods  are  needed  to  ensure  instructor 
agreement  on  grading  criteria  and  to  provide  specific  information  to  teams  on  perfor¬ 
mance  requirements  and  training  achievement. 

Purpose 

The  purpose  of  this  effort  was  to  evaluate  the  importance  of  team  communication 
measures  for  assessing  performance  of  antisubmarine  warfare  (ASW)  teams  and  subteams 
during  team  training  exercises. 

Approach 

An  initial  study  was  conducted  to  identify  useful  criterion  measures  of  team 
performance.  Data  were  gathered  on  ASW  teams  from  escort  ships  who  were  undergoing 
qualification  training  in  the  14A2  ASW  single  ship  team  trainer.  The  ASW  subteams 
included  were  those  in  the  combat  information  center,  underwater  battery  plot,  sonar,  and 
bridge.  Subteam  and  individual  performance  and  weapon  accuracy  were  assessed  by 
instructors  using  a  detailed  checklist. 

Next,  published  data  on  communications  of  ASW  helicopter  crews  were  analyzed  to 
evaluate  the  feasibility  of  using  communications  measures  for  assessing  performance.  A 
revised  system  for  classifying  communications  was  developed.  This  system,  which 
classifies  messages  into  information,  questions,  responses  to  questions,  affirmations 
(repetition)  of  information,  evaluations,  directions,  and  other,  was  applied  to  data 
collected  from  14A2  exercises  for  16  ships'  teams.  Recordings  were  made  of  voice 
transmissions  over  the  1JS  (command),  613S  (sonar),  R/T2  (ship  to  ship)  and  R/T3  (ship  to 
air)  circuits.  Message  rates  (amount  of  information  per  minute)  were  determined  for 
different  types  of  messages  and  compared  to  instructor  scores  far  individuals  and 
subteams. 

Findings 

In  the  initial  study,  the  checklist  appeared  to  be  sensitive  to  team  effectiveness,  as 
indicated  by  improvement  over  trials,  but  considerable  difficulties  were  found  in  its  use. 
Also,  weapon  use  was  found  to  be  an  unsatisfactory  measure  of  team  performance  because 
of  wide  variation  in  number  of  weapons  fired  and  lack  of  relationships  with  other 
performance  measures. 

In  the  communications  study,  a  number  of  significant  correlations  were  found 
between  message  rates  and  performance  scores,  with  the  higher  correlations  tending  to 
occur  for  external  communications  (on  the  RT/2  and  RT/3  circuits)  and  for  the  communi¬ 
cation  categories  of  information,  responses  to  questions,  and  evaluations.  These  relation¬ 
ships  generally  do  not  seem  to  be  strong  enough  for  communication  rates  to  be  used  as 
direct  measures  of  performance. 

Conclusions 

Instructor  grades  appear  to  be  related  to  team  communication  rates  but  quantitative 
measures  of  team  communications  are  not  useful  by  themselves  for  assessing  team 


performance.  Quality  and  timeliness  of  communications  appear  to  be  more  appropriate 
measures  for  scoring  team  and  individual  performance. 

A  composite  scoring  system,  sensitive  to  the  multiple  events  occurring  in  A5W 
exercises,  is  needed  for  assessment  and  feedback  during  ASW  team  training.  For 
maximum  objectivity  and  accuracy,  the  system  should  employ  a  checklist  that  can  be 
completed  during  the  course  of  an  exercise. 

Recommendations 

1.  An  objective  performance  checklist  should  be  developed  for  use  in  14A2  ASW 
training  exercises  and  for  grading  Phase  I  qualification  exercises.  Particular  attention 
should  be  given  to  assessing  the  timeliness  and  quality  of  communications. 

2.  Research  and  development  should  be  conducted  to  provide  the  necessary 
information  to  develop  a  performance  checklist  that  accurately  reflects  training 
objectives.  Data  taken  from  existing  tapes  of  communications  collected  for  this  report 
should  be  used  to  assist  in  developing  the  objectives  and  checklist. 
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INTRODUCTION 


Problem 


Performance  assessment  in  most  current  team  training  is  highly  subjective  (Hall  A 
Rizzo,  1975;  Wagner,  Hibbits,  Rosenblatt,  A  Schutz,  1976).  Hall  and  Rizzo  concluded  that 
"clearly  stated,  objective  criteria  and  procedures  for  evaluating  team  performance  are 
apparently  not  available"  (p.  43).  Objective  methods  are  needed  to  ensure  instructor 
agreement  on  grading  criteria  and  to  provide  detailed  feedback  to  teams  and  team 
members.  In  the  area  of  antisubmarine  warfare  (ASW),  there  are  two  key  problems  in  the 
objective  assessment  of  team  performance:  (1)  the  development  of  appropriate  measures 
of  team  communications,  and  (2)  the  determination  of  useful  overall  measures  of  team 
performance. 

Purpose 

The  primary  objective  of  this  effort  was  to  evaluate  the  usefulness  of  quantitative 
measures  of  the  volume  of  different  types  of  communications  in  surface  ship  ASW  training 
exercises.  Supporting  objectives  were  (1)  to  evaluate  existing  criterion  measures  of  team 
performance,  and  (2)  to  develop  a  system  for  classifying  communications  and  measuring 
their  volume. 

Background 

This  work  is  part  of  a  program  to  develop  objective  performance  measures  for  team 
training.  The  project  examined  team  performance  during  exercises  in  the  14A2  single¬ 
ship  ASW  team  trainer.  The  14 A2  trainer  is  used  during  Phase  I  of  the  Escort 
Qualification  Program  to  qualify  ships  for  escort  duty. 

An  objective  performance  measurement  system  has  a  number  of  requirements.  A 
measure  of  overall  team  performance  such  as  weapon  accuracy  is  not  enough  by  itself  for 
assessing  14A2  team  performance  because  of  the  complexity  of  the  exercises.  A 
composite  scoring  system  is  needed  that  reflects  the  responses  of  individuals  and 
subteams  to  all  of  the  critical  events  in  an  ASW  problem.  The  system  must  provide 
extensive  coverage  of  communications  because  communications  are  essential  to  ASW 
operations  and  are  of  high  frequency.  In  earlier  work  in  this  project,  Bell  (in  press) 
identified  a  number  of  poterrtratty  useful  objective  measures  of  performance  in  14A2 
training.  Most  of  these  measures  were  concerned  with  correctness  of  procedures  and  time 
and  accuracy  of  team  and  member  actions.  There  were  few  available  measures  of  team 
communications.  Bell  did  find  that  frequency  of  communication  errors  by  the  tactical 
communicator  (T ACCOM)  was  negatively  correlated  with  instructor  grades  for  the 
TACCOM. 

Previous  attempts  to  find  important  quantitative  measures  of  communications  have 
been  only  moderately  successful.  It  appears  that  overall  communication  rates  are 
unrelated  to  team  performance  (Siskel,  1965;  Fineburg,  1974).  There  have  been  several 
attempts  to  find  relationships  between  team  effectiveness  and  message  rates  in  various 
subcategories  of  communications.  Krumm  and  Farina  (1962),  in  a  study  of  simulated  B-52 
bombing  missions,  identified  four  principal  communication  categories  (other  than 
identifications  and  acknowledgements):  requests,  responses  to  requests,  voluntary  com¬ 
munications,  and  orders.  They  found  that  voluntary  communications  increased  with 
training  and  were  directly  related  to  bombing  and  navigation  accuracy  and  to  instructor 
rankings  of  crew  coordination. 


Briggs  and  Johnston  (1966,  1967)  investigated  the  communications  of  air  intercept 
controllers  in  simulated  antiair  warfare.  Using  results  of  a  factor  analysis,  they  identified 
three  major  communication  categories:  (1)  declarative  messages,  based  on  information 
from  the  radar  scope,  (2)  tactical  messages,  based  on  information  not  on  the  scope,  and  (3) 
tactical  commands.  In  a  series  of  experiments,  Briggs  and  Johnston  found  few  significant 
correlations  between  team  performance  and  volume  of  communications  in  these  catego¬ 
ries.  Most  of  the  significant  relationships  were  negative,  indicating  a  need  for  restraint  in 
communications.  Only  where  communications  appeared  to  be  essential  (e.g.,  when 
information  on  the  scope  was  restricted)  was  there  a  positive  relationship. 

Federman  and  Siegel  (1965)  analyzed  communications  during  ASW  helicopter  mis¬ 
sions.  These  data  are  important  because  they  deal  with  ASW  and  because  Federman  and 
Siegel  employed  a  number  of  communication  measures  not  used  by  previous  investigators. 
They  performed  a  factor  analysis  of  the  measures  and  found  four  factors  related  to 
weapon  miss  distance.  They  labeled  the  factors  (1)  probabilistic  structure  (messages 
reflecting  uncertainty  and  the  weighing  of  probabilities),  (2)  evaluative  interchange,  (3) 
hypothesis  formulation,  and  (4)  leadership  control.  However,  their  analysis  is  suspect 
because  they  used  a  number  of  variables  that  were  ratios  of  other  variables  within  the 
analysis.  This  amounts  to  giving  these  variables  an  arbitrarily  high  weight  in  the  factor 
analysis.  A  more  appropriate  analysis  is  needed  to  identify  useful  categories  of  ASW 
com  munications. 

Federman  and  Siegel  (1965)  also  measured  time  to  detect  the  submarine,  time  to 
initiate  changes  in  sonar  dip  location,  dip  to  dip  accuracy,  and  weapon  miss  distance  in 
exercises  of  varying  difficulty.  Only  miss  distance  was  related  to  difficulty.  The 
usefulness  of  miss  distance  for  surface  ship  ASW  needs  to  be  checked. 


APPROACH 

1.  Criterion  performance  measures  available  on  the  14A2  team  trainer  were 
analyzed  to  identify  useful  measures  of  team  and  subteam  performance.  Such  measures 
are  needed  not  only  to  determine  team  readiness  but  also  to  evaluate  the  validity  of 
proposed  performance  measures  such  as  communication  rates. 

2.  The  Federman  and  Siegel  data  were  reanalyzed  to  identify  useful  measures  of 
ASW  communication  rates. 

3.  A  study  was  made  of  ASW  communication  rates  and  team  performance  in  14A2 
exercises  to  determine  whether  quantitative  measures  of  ASW  communications  could  be 
used  for  performance  evaluation. 


PRELIMINARY  ANALYSIS  OF  CRITERION  MEASURES 


ASW  Exercises 


The  14A2  exercises  studied  were  search-attack  unit  (SAU)  exercises.  SAU  exercises 
provide  a  simulation  of  the  crew's  own  ship,  a  high  value  unit  (HVU)  that  they  are  assigned 
to  protect,  an  assist  ship  with  which  their  ship  must  operate,  one  or  two  helicopters  used 
to  support  the  operations,  and  an  enemy  submarine  whose  primary  goal  is  to  attack  the 
HVU.  The  HVU  and  assist  ship,  the  two  aircraft,  and  the  submarine  are  maneuvered  by 
instructors.  Ship  and  aircraft  maneuvers,  weapon  use,  etc.,  occur  in  a  free  play  mode 
starting  from  preprogrammed  initial  positions.  Communications  between  crew  members,  the 
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HVU,  the  support  ship,  and  the  aircraft  simulate  actual  operations.  The  instructors  act  as 
the  communicators  for  the  external  units. 

SAU  exercises  provide  training  for  crew  members  in  the  combat  information  center 
(CIC),  sonar,  underwater  battery  (UB)  plot,  and  bridge.  Personnel  in  CIC  include  the 
evaluator,  who  directs  ASW  operations;  the  tactical  communicator  (TACCOM),  who 
communicates  with  the  other  ships;  the  antisubmarine  air  controller  (ASAC),  who  directs 
ASW  aircraft  operations;  geographic  plotters;  and  a  few  others.  Sonar  includes  the  sonar 
supervisor  and  operators.  Personnel  in  UB  plot  are  the  ASW  officer  and  operators. 

Procedure 


Instructor  grade  sheets  were  collected  for  12  ships'  teams  taking  the  3-day  escort 
qualification  course  in  the  14A2  trainer.  Instructors  used  a  checklist  to  score  the 
exercises.  The  checklist  provided  evaluation  of  specific  actions  taken  by  team  members 
or  subteams.  Scores  on  the  separate  items  were  added  for  each  team  member  and  then 
weighted  to  determine  the  team  score.  The  weights  used  were  as  follows: 

1.  Evaluator — 40  percent 

2.  ASAC-- 16  percent 

3.  CIC— 12  percent 

4.  Sonar — 8  percent 

5.  UB  plot-- 12  percent 

6.  Bridge—2  percent 

7.  Weapons  use — 10  percent 


Scores  for  teams  and  individual  positions  were  examined  for  exercise  5,  exercise  12, 
and  the  final  grade.  Exercise  5  used  a  SAU  composed  of  own  ship,  assist  ship,  and  a 
Lamps  helicopter.  Exercise  12  added  a  sonar  dipping  helicopter  to  the  SAU.  In  the  final 
grade,  the  last  exercise  counted  40  percent,  and  the  previous  exercises  together  counted 
60  percent.  At  the  time  of  this  study,  there  were  usually  12  exercises,  although  some  of 
the  initial  exercises  might  not  be  graded. 

Results 


The  weighted  instructor  scores  for  checklist  items  were  converted  to  percentages  to 
permit  comparisons  between  individuals  and  subteams.  Results  are  shown  in  Table  1. 

An  important  feature  of  the  checklist  was  that  it  provided  useful  information  on 
performance  at  a  variety  of  levels:  teams,  subteams,  individual  positions,  and  specific 
subteam  or  member  actions.  For  example,  since  sonar  and  UB  plot  operators  scored 
higher  than  all  other  individuals  and  subteams  on  trial  5,  it  appears  that  these  operators 
tend  to  reach  satisfactory  proficiency  levels  more  quickly  than  other  team  members. 
Other  team  members,  except  for  the  ASAC,  showed  large  improvements  in  performance 
from  exercise  5  to  the  final  exercise.  Most  of  these  improvements  were  statistically 
significant  (t  for  paired  measures),  despite  the  small  sample  size.  In  some  cases,  the 
standard  deviation  decreased,  indicating  greater  uniformity  among  ships. 

The  contributions  of  subteam  scores  to  team  scores  are  shown  by  the  correlations  in 
the  top  part  of  Table  2.  Correlations  were  highest  for  the  evaluator,  as  might  be 
expected  since  his  score  counted  40  percent  of  the  team  score.  Sonar  and  UB  plot  scores 
were  not  significantly  correlated  with  team  score  on  either  exercise  5  or  the  final  grade. 
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Table  i 


Percentage  Score  on  Instructor  Checklist 
in  Preliminary  Study 


Exercise  5 

Final  Exercise 

Item 

Mean 

S.D. 

Mean 

S.D. 

Evaluator 

General 

58.8 

16.8 

79.9** 

13.7 

Approach 

73.5 

17.5 

85.7 

13.5 

Attack 

73.8 

22.6 

79.9 

18.8 

ASAC 

76.5 

15.1 

80.0 

10.0 

CIC 

TACCOM 

76.8 

18.5 

90.4** 

9.0 

Other 

74.3 

12.2 

90.3** 

6.3 

Sonar 

Supervisor 

69.7 

15.7 

83.1** 

14.6 

Operators 

84.5 

12.5 

89.3 

14.0 

UB  Plot 

ASW  Officer 

73.2 

11.4 

86.2* 

10.0 

Operators 

85.5 

20.1 

96.0 

7.4 

Bridge 

79.6 

15.7 

93.1 

12.8 

Weapons 

73.0 

29.6 

58.3 

36.9 

Average 

74.9 

— 

84.4 

— 

*p  <  .05. 

**p  <  .01. 


Table  2 


Correlations  of  Subteam  Scores  with  Criterion  Scores 

in  Preliminary  Study  , 

i 

i 

. — — . —  . -  . .  —  —  ■■■  . ■■■■  —  —  \ 

Subteam 


Criterion 


Score 

Eval. 

ASAC 

C1C 

Sonar 

UB  Plot 

Bridge 

Team  Totals 

Exercise  5 

.88** 

.63* 

.80** 

.04 

.32 

.32 

Final  Grade 

.85** 

.73** 

.78** 

.48 

-.30 

.86** 

Evaluator: 

Exercise  5 

— 

.23 

.53 

-.11 

.32 

.11 

Final  Grade 

— 

.69* 

.59* 

.59* 

-.45 

.65* 

Weapons  Use: 

Exercise  5 

.49 

.37 

.52 

-.47 

-.06 

.49 

Final  Grade 

.23 

.35 

.45 

-.06 

-.33 

.56 

*p  <  .05. 

**p  <  .01. 


Subteam  scores  tended  to  be  poorly  correlated  with  each  other  except  that  there  was 
a  strong  tendency  for  evaluator  scores  to  be  correlated  with  other  subteam  scores  on  the 
final  grade.  The  evaluator  is  the  key  person  on  the  ASW  team.  The  data  suggest  that 
performance  on  the  better  teams  is  partly  due  to  better  coordination  of  team  activities  by 
the  evaluators. 

No  significant  correlations  were  found  between  weapons  use  scores  and  any  of  the 
scores  for  subteams.  The  weapons  score  was  based  on  miss  distance  of  both  ASROC  and 
directed  helicopter  attacks.  A  ship  might  attack  from  one  to  foir  times  in  an  exercise, 
depending  on  both  tactical  conditions  and  team  skill.  Occasionally,  a  ship  might  not  be 
able  to  fire  at  all.  Thus,  weapon  accuracy  seems  to  be  of  minor  importance  for  assessing 
team  performance  in  the  14A2  trainer. 

An  analysis  of  the  content  of  the  checklist  revealed  that  half  of  the  126  items  dealt 
partially  or  totally  with  communications,  and  a  number  of  other  items  also  reflected  the 
great  importance  of  team  communications  in  ASW  operations.  Most  of  the  checklist 
items  were  objective  in  nature,  requiring  only  comparison  against  a  known  standard  (e.g., 
"Was  Doppler  reported  accurately?").  Some  items  depended  heavily  on  expert  judgment 
(e.g.,  "Did  the  evaluator  keep  firm  control  of  and  order  correct  tactics  for  all  air 
assets?”).  Over  half  of  the  evaluator  grade  depended  on  this  type  of  item.  Checklists 
were  filled  out  at  the  end  of  the  exercise,  a  practice  that  tends  to  reduce  reliability  and 
objectivity.  It  is  highly  desirable  to  have  an  objective  system  where  performance 
standards  are  written  down  and  evaluation  is  done  during  an  exercise.  Such  a  system  is 
advantageous,  since  it  ensures  instructor  agreement  on  evaluation  criteria,  is  useful  for 
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training  of  new  instructors,  and  can  be  used  to  provide  explicit  information  to  teams  on 
what  is  expected  of  them. 

In  summary,  both  team  and  subteam  scores  on  the  checklist  appear  to  be  important 
criterion  measures  for  evaluating  the  usefulness  of  proposed  communication  measures. 
Weapon  accuracy,  however,  does  not  seem  to  be  useful  for  the  very  complex  ASW 
exercises  studied  here. 


DEVELOPMENT  OF  CLASSIFICATION  SCHEME  FOR  SURFACE 
SHIP  ASW  COMMUNICATIONS 


Method 


Federman  and  Siegel  collected  communication  data  for  24  ASW  helicopter  crews, 
developed  27  measures  of  communication  rates,  and  used  the  14  measures  that  correlated 
most  highly  with  weapon  miss  distance  for  their  factor  analysis.  Fourteen  is  a  rather 
large  number  of  variables  for  a  factor  analysis  based  on  data  from  only  24  crews. 
Firther,  several  measures  were  redundant  and  others  were  formed  by  ratios  of  other 
variables,  which  gave  undue  weight  to  these  measures.  Therefore,  the  Federman  and 
Siegel  data  were  reanalyzed,  using  a  principal  factors  equimax  rotation,  to  reduce 
redundancy  among  variables  and  keep  the  number  of  variables  to  a  reasonable  size. 
Derived  variables,  such  as  the  ratio  of  two  measured  variables,  were  eliminated  to  give 
appropriate  weight  to  the  original  measures.  Federman  and  Siegel  also  classified 
individual  communications  in  more  than  one  way  (e.g.,  "Information  Response"  and 
"Phenomenological"),  giving  additional  weight  to  these  communications.  Some  of  these 
dual  classifications  were  eliminated  on  the  basis  of  a  partial  correlation  analysis.  Finally, 
four  measures  were  eliminated  that  occurred  less  than  1/2  of  1  percent  of  the  time.  As  a 
result,  only  nine  variables  were  included  in  the  new  factor  analysis. 

Results 


Factor  analysis  of  the  nine  communication  variables  selected  from  the  Federman  and 
Siegel  data  resulted  in  the  identification  of  five  factors,  which  are  presented  in  Table  3. 
The  Eigenvalues  and  percent  of  variance  figures  indicate  the  relative  importance  of  the 
factors.  The  makeup  of  the  first  four  factors  suggested  that  they  should  be  labelled 
direction,  data  assimilation,  data  acquisition,  and  prediction  (or  evaluation).  The  fifth 
factor  is  of  lesser  importance  but  is  most  closely  associated  with  responses  to  requests  for 
information. 

Factor  1,  direction,  covers  messages  expressing  a  need  for  timely  action,  such  as 
orders  and  recommendations.  The  variables  of  activity  and  extrapolation  have  the 
strongest  loadings  on  this  factor.  Examples  given  by  Federman  and  Siegel  indicate  that 
messages  in  these  categories  often  have  a  strong  directive  component  (e.g.,  activity: 
"Recommend  you  jump  east  now";  extrapolation:  "The  sub's  going  south  at  10  knots,  so  fly 
south  at  90  knots  and  we'll  be  over  him  in  62  seconds”). 

Factor  2,  data  assimilation,  reflects  the  need  for  the  team  members  to  absorb  the 
meaning  in  data.  The  principal  contributing  variables  are  corroboration  ("I  understand, 
080  break  22"),  voluntary  information,  and  interpolation. 

Factor  3,  data  acquisition,  covers  messages  that  emphasize  the  need  to  fill  in  gaps  in 
existing  data.  The  major  variables  are  requests  for  information  and  interpolation. 
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Table  3 


Factor  Loadings  for  Federman  and  Siegel  Data 


Variable 

Direction 

1 

Data 

Assimilation 

2 

Data 

Acqusition 

3 

Prediction 

or 

Evaluation 

4 

Requests 

for 

Information 

5 

Activity 

.89 

.05 

.06 

-.04 

-.02 

Corroboration 

.15 

.73 

.18 

.07 

.02 

Request  Infor¬ 
mation 

.12 

-.59 

.75 

.21 

.06 

Information 

Response 

-.08 

-.02 

O 

• 

i 

-.03 

.71 

Voluntary 

Information 

-.07 

.36 

-.02 

-.10 

-.02 

Voluntary 

Opinion 

-.04 

-.05 

.03 

.83 

.01 

Interpolation 

-.45 

.39 

.60 

-.05 

.24 

Extrapolation 

.41 

o 

© 

• 

-.15 

.39 

.56 

Repeated  Message 

.03 

.07 

.28 

O 

• 

1 

-.16 

Eigenvalue 

1.45 

1.25 

1.06 

.87 

.65 

%  of  Variance 

28 

23 

20 

17 

12 

Factor  4,  prediction  or  evaluation,  covers  messages  that  predict  events  or  evaluate 
the  outcome  of  actions.  This  factor  loads  most  heavily  on  voluntary  opinion  and,  to  a 
lesser  extent,  on  extrapolation.  Note  that  the  example  of  voluntary  opinion  given  in 
Federman  and  Siegel,  "He  should  be  due  north  of  you,"  could  also  be  classified  as 
extrapolation. 

The  Federman  and  Siegel  procedure  of  classifying  messages  in  more  than  one 
category  seems  to  be  counterproductive.  For  instance,  the  definitions  and  examples  of 
interpolation  and  extrapolation  indicate  that  messages  in  -these  categories  could  be 
classified  in  a  number  of  other  ways.  In  the  factor  analysis,  interpolation  and 
extrapolation  were  not  simply  related  to  the  performance  criterion  but,  instead,  each 
loaded  on  several  factors.  On  the  basis  of  this  analysis,  the  set  of  seven  categories  listed 
below  was  developed  for  analyzing  ASW  communications.  Six  of  the  categories  are 
similar  to  the  variables  in  the  factor  analysis  that  loaded  primarily  on  only  one  factor 
(voluntary  information,  requests,  responses,  corroboration,  voluntary  opinion,  and 
activity).  The  categories  have  the  desirable  property  of  being  mutually  exclusive  and 
exhaustive  (when  applied  to  individual  message  units  rather  than  the  total  message). 

1.  Information.  Messages  that  transmit  information  essential  to  keeping  track  of 
events  or  controlling  operations  (e.g,  "Target  bears  145,  at  8500  yards").  This  category 


covers  "voluntary"  information  and  does  not  include  responses  to  questions  or  affirmations 
of  received  data. 

2.  Questions.  Requests  for  information,  intentions,  opinions,  etc.  (e.g.,  "What  is 
your  present  position?"). 

3.  Responses.  Answers  to  questions. 

4.  Affirmations.  Repetition  of  information  received  (e.g.,  "Roger.  Read  your 
bearing  140,  range  7900  yards").  Classified  as  "Other"  if  the  information  is  repeated 
incorrectly. 

3.  Evaluations.  Messages  that  evaluate  the  tactical  situation  or  some  phase  of  it 
(e.g.,  "I  hold  the  target  headed  northwest  at  slow  speed").  An  immediately  following,  "I 
concur,"  would  also  be  classified  as  evaluation. 

6.  Directions.  Directions  regarding  actions,  including  orders,  commands,  and  strong 
suggestons  that  have  the  effect  of  directing  (e.g.,  "Recommend  you  come  right  to  040 
degrees"). 

7.  Other.  Messages  that  cannot  be  placed  in  the  other  six  categories,  including  call 
signs,  irrelevant  information,  facetious  remarks,  etc. 

This  system  for  categorizing  communications  was  used  in  the  analysis  to  identify 
relationships  between  communication  rates  and  team  performance.  During  this  analysis, 
it  was  found  that  message  units  could  be  classified  consistently  in  these  categories  with  a 
relatively  small  amount  of  practice. 


IDENTIFICATION  OF  RELATIONSHIPS  BETWEEEN  COMMUNICATION 
RATES  AND  TEAM  PERFORMANCE 


Subjects 

The  subjects  were  16  crews  taking  the  1070  qualification  course  or  the  1032  training 
course  in  the  14A2  trainer.  The  makeup  of  the  crews  was  highly  variable,  some  members 
having  had  no  previous  experience  in  team  ASW  operations.  Four  of  the  14  crews 
scheduled  for  the  1070  course  were  considered  to  be  unprepared  by  the  instructors,  and 
they  were  changed  to  the  1032  training  courses.  In  addition,  data  for  two  crews 
voluntarily  doing  1032  training  were  added  to  increase  the  sample  size.  The  1032  and 
1070  courses  used  basically  the  same  exercises  and  were  scored  in  the  same  way  by  the 
instructors. 

Procedure 


During  this  study,  the  1070  course  was  2  days  in  length  and  covered  seven  exercises. 
Exercises  3  and  7  were  selected  for  study  to  measure  changes  in  performance.  Exercise  3 
included  an  assist  ship,  a  Lamps  helicopter,  and  an  enemy  submarine.  Exercise  7  added  a 
sonar  dipping  helicopter  but  was  otherwise  essentially  the  same.  The  1032  exercises  that 
were  recorded  had  the  same  scenerios  as  these  exercises. 

A  four-channel  tape  recorder  was  used  to  record  communications  over  the  four 
critical  communication  networks:  (1)  the  13S,  the  principal  command  channel  aboard  ship, 
(2)  the  613S,  the  sonar  reporting  network,  (3)  the  R/T2  radio  transmission  net,  linking  the 
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ship  to  the  support  unit  and  high  value  unit,  and  (4)  the  R/T3  net,  linking  the  ship  with  the 
support  aircraft. 

Measures 


Communications  were  divided  into  distinct  message  units.  A  message  unit  was 
defined  as  that  portion  of  a  communication  that  transmitted  a  single  item  of  data.  For 
example,  the  bearing  to  a  target,  an  identifying  call  sign,  particular  descriptive  data,  such 
as  "my  latest,"  etc.,  were  classified  as  individual  message  units.  Each  message  unit  was 
classified  into  one  of  the  seven  categories  obtained  from  the  reanalysis  of  the  Federman 
and  Siegel  data  (i.e.,  information,  questions,  responses  to  questions,  affirmations,  evalua¬ 
tions,  directives,  or  other). 

Audio  tapes  were  made  for  each  ship,  and  then  played  back  and  "read"  to  count  the 
number  of  message  units  in  each  communication.  For  consistency,  all  tapes  were  read  by 
one  individual.  The  number  of  message  units  in  each  category  was  divided  by  the  duration 
of  the  exercise  to  establish  communication  rates,  except  that  the  duration  of  sonar 
contact  was  used  to  establish  communication  rates  over  the  610S  net.  These  data  were 
then  correlated  with  instructor  grades. 

The  grading  system  was  different  from  that  used  in  the  preliminary  study.  The 
checklist  was  no  longer  used  and  separate  scores  for  the  bridge  and  weapons  use  were  no 
longer  computed.  At  the  end  of  each  exercise,  the  instructors  assigned  subjective  scores 
to  the  evaluator,  ASAC,  and  TACCOM,  and  to  the  CIC,  sonar,  and  UB  plot  subteams  by 
awarding  a  portion  of  a  specified  maximum  score  (e.g.,  40  for  the  evaluator).  These 
scores  were  added  together  to  produce  a  team  exercise  score  (maximum  =  100).  Final 
grades  were  established  by  averaging  the  scores  for  all  seven  exercises.  A  final  grade  of 
62.5  was  required  for  qualification. 

Results 

The  mean  number  of  message  units  transmitted  per  minute  over  the  four  communi¬ 
cation  circuits  included  in  this  analysis  are  shown  in  Table  4.  There  was  a  wide  variation 
in  message  rates  in  the  various  communication  categories.  Rates  were  highest  in  the 
information  category,  as  might  be  expected  where  the  activities  of  several  ships,  aircraft, 
and  subteams  must  be  coordinated.  Directions  were  also  frequent  except  over  the  61 JS 
circuit,  which  is  basically  a  reporting  circuit  for  sonar . 

For  the  RT/3  air  control  circuit,  there  was  a  significant  increase  in  volume  of 
communications  from  exercise  3  to  exercise  7  in  all  categories  except  affirmations  and 
evaluations.  The  reason  for  this  increase  is  that  the  ASAC  directed  two  aircraft  in 
exercise  7  but  only  one  in  exercise  3.  Rates  on  the  other  circuits  were  about  the  same  in 
the  two  exercises  except  for  information  passed  on  the  RT/2  intership  circuit.  Instructor 
grades  may  be  negatively  related  to  this  increase  on  the  RT/2  circuit,  as  discussed  later. 

Instructor  grades  were  used  as  the  criteria  for  evaluating  communications  measures. 
Percentage  scores  for  teams  and  subteams  are  shown  in  Table  5.  Instructor  grades  may  be 
obtained  from  the  percent  scores  by  using  the  weights  at  the  left  of  Table  5.  The 
evaluator  score  counts  40  percent  of  the  team  grade.  Since  this  weighting  tends  to 
obscure  some  of  the  differences  between  teams,  it  is  important  to  look  at  both  team  and 
subteam  scores  in  relating  communications  to  performance.  For  example,  on  exercise  3, 
the  ASACs  were  much  more  variable  in  performance  from  team  to  team  than  were  the 
evaluators.  This  is  indicated  by  the  standard  deviation  of  the  unweighted  percentage 
scores  (8.0  for  the  evaluator  and  18.7  for  the  ASAC).  However,  when  the  scores  are 
weighted  (40%  and  15%)  for  assigning  team  grades,  the  standard  deviation  for  evaluators 
is  larger  (3.2  vs.  2.8). 
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Table  4 


Communication  Rates  in  Exercises  3  and  7 
(Number  of  Message  Units  Per  Minute) 


13S 

61 JS 

R/T2 

Rm 

Command 

Sonar 

Ship  to  Ship 

Ship  to  Air 

Category  Ex. 

Mean 

S.D. 

Mean 

S.D. 

Mean 

S.D. 

Mean 

S.D. 

Information 

3 

3.44 

1.18 

7.01 

2.23 

2.86 

.74 

2.76 

.85 

7 

3.56 

1.36 

7.30 

3.47* 

.84 

4.11* 

.94 

Questions 

3 

1.11 

.48 

.34 

.34 

.14 

.21 

.13 

7 

.88 

.56 

.46 

.53 

.15 

.12 

.46* 

.28 

Responses 

3 

1.05 

.49 

.34 

.44 

.20 

.15 

.21 

.15 

7 

.88 

.48 

.41 

.42 

.12 

.10 

.52* 

.45 

Affirmations 

3 

.64 

.41 

.02 

.06 

.02 

.04 

.10 

7 

.57 

.27 

.00 

.00 

.12 

.16 

Evaluations 

3 

.56 

.07 

.14 

.16 

.11 

.14 

7 

.70 

.62 

.09 

.18 

.24 

.14 

.08 

.09 

Directions 

3 

1.91 

.48 

.08 

.24 

.83 

1.07 

.57 

7 

1.84 

.46 

.06 

.10 

2.40 

.77 

2.53* 

.48 

Other 

3 

3.70 

.79 

.78 

1.19 

2.95 

.60 

2.22 

.72 

7 

3.34 

1.03 

.21 

.30 

3.28 

.69 

4.26* 

.78 

*p  <  .03. 

Table  5 

Percentage  Scores  on  Exercise  3  and  7 

Exercise  3 

Exercise  7 

Subteam 

Mean 

S.D. 

Mean 

S.D. 

Evaluator  (40,40) 

72.2 

8.0 

76.8 

11.5 

ASAC  (15,15) 

67.3 

18.7 

67.3 

16.4 

TACCOM  (15,15) 

64.0 

13.3 

74.0* 

12.3 

C1C  (15,10) 

75.3 

12.3 

77.0 

8.0 

Sonar  (10,10) 

69.0 

10.7 

76.0* 

8.3 

UB  Plot  (5,10) 

66.0 

14.6 

77.0* 

14.0 

Average 

69.0 

— 

74.7 

— 

Team  Grade  (100) 

70.0 

— 

74.9 

— 

Note.  Numbers  in  parentheses  are  weights  (%)  used  for  grading  in  exercises  3  and  7 
respectively. 


*p  <  .03. 
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Mean  percentage  scores  for  the  TACCOM,  sonar,  and  UB  plot  were  significantly 
better  on  exercise  7  than  on  exercise  3.  This  is  probably  a  straightforward  practice  effect 
due  to  the  fact  that  these  groups  performed  essentially  the  same  tasks  on  both  exercises. 
In  contrast,  the  ASACs  faced  a  considerably  more  complex  task  on  exercise  7,  and  their 
mean  percentage  score  did  not  change. 

The  standard  deviations  in  Table  5  suggest  that  most  of  the  variation  in  team  grades 
is  attributable  to  variations  in  grades  for  the  evaluator,  ASAC,  and  TACCOM.  A  similar 
result  is  obtained  if  correlations  are  computed  between  team  and  subteam  grades- -see 
Table  6.  Evaluator  grades  were  most  highly  correlated  with  team  scores.  The  ASAC  and 
TACCOM  scores  were  the  only  other  scores  that  were  consistently  correlated  with  team 
grades  on  exercises  3  and  7.  This  is  understandable  since  the  C1C,  sonar,  and  UB  plot 
scores  contributed  at  most  10  percent  each  to  the  team  grade. 


Table  6 

Correlations  Between  Subteam  Scores  and  Criterion  Scores 


Subteam 


Criterion 


Score 

Eval 

ASAC 

TACCOM 

CIC 

Sonar 

UB  Plot 

Team  Total: 

Exercise  3 

.74** 

.60* 

.46* 

.46* 

.43 

.05 

Exercise  7 

.89** 

.74** 

.44* 

.24 

.55* 

.48* 

Evaluator: 

Exercise  3 

— 

.18 

.01 

.46* 

.20 

-.09 

Exercise  7 

— 

.46* 

.30 

.19 

.50* 

.26 

*p  <  .05. 

**p  <  .01. 


There  were  very  few  significant  correlations  between  subteam  scores.  There  was  a 
slight  tendency  for  evaluator  scores  to  be  related  to  the  scores  of  other  team  members 
but  the  relationship  was  not  nearly  as  marked  as  in  the  preliminary  study.  The  checklist 
measured  performance  at  a  finer  level  of  detail  and  so  may  have  been  more  sensitive  to 
subteam  dependencies. 

Correlations  were  computed  between  instructor  grades  and  communication  rates  to 
see  if  rates  in  the  various  communication  categories  were  related  to  performance. 
Correlations  were  obtained  separately  for  each  of  the  four  communication  networks  and 
are  shown  in  Table  7.  Overall,  relatively  few  significant  correlations  were  found.  Most  of 
the  significant  correlations  occurred  for  the  external  ship-to-ship  and  ship-to-air  circuits, 
and  were  concentrated  most  heavily  in  the  communication  categories  of  information, 
responses  to  questions,  and  evaluations. 

The  pattern  of  correlations  changed  noticeably  from  exercise  3  to  exercise  7  for  all 
circuits.  On  the  R/T3  ship-to-air  circuit,  the  correlations  tended  to  shift  from  negative 
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to  positive  values  but,  on  each  of  the  other  three  circuits,  the  correlations  tended  to  shift 
toward  negative  values;  that  is,  positive  correlations  became  smaller  or  negative  and 
negative  correlations  became  more  so.  These  results  suggest  either  that  the  instructors 
graded  differently,  that  the  better  teams  learned  to  reduce  unnecessary  communications, 
or  both.  Information  on  this  point  may  be  obtained  by  a  more  detailed  examination  of 
correlations  on  the  external  circuits,  where  the  clearest  trends  occurred. 

On  the  R/T2  intership  circuit,  no  particular  pattern  was  evident  on  exercise  3; 
however,  on  exercise  7,  most  of  the  correlations  were  negative.  All  of  the  correlations 
for  the  "voluntary"  message  categories  of  information  and  evaluations  were  negative  and 


1  3  six  of  these  correlations  were  significant.  Discussion  with  instructors  revealed  that  the 


1  reason  for  these  negative  relationships  is  that  instructors  downgrade  unnecessary  com- 

I  ’  munications.  Particular  mention  was  made  of  the  fact  that  weaker  teams  often  overload 

1  the  RT/2  net  by  making  too  many  status  reports  to  the  task  group  commander.  It  may  be 

r  noted  that  all  of  the  significant  negative  correlations  for  subteams  on  trial  7  occurred  for 

i  the  TACCOM  and  evaluator,  who  together  control  the  message  traffic  on  this  net. 


On  the  R/T3  air  control  circuit,  there  was  a  mild  tendency  on  exercise  3  for  message 
rates  to  be  negatively  related  to  performance  scores.  This  tendency  was  clearly  reversed 
on  exercise  7  where,  except  for  directions,  the  great  majority  of  the  correlations  were 
positive.  Most  of  the  significant  correlations  occurred  for  the  ASAC  and  evaluator.  On 
exercise  7,  the  ASAC  and  evaluator  were  responsible  for  controlling  two  aircraft  rather 
than  one,  and  this  necessitated  a  great  increase  in  RT/3  communications  (see  Table  4). 
These  results,  together  with  the  results  for  the  RT/2  circuit,  are  comparable  to  the 
findings  of  Briggs  and  Johnston  (1966,  1967)  that  relationships  between  communication 
rates  and  team  performance  tend  to  be  positive  for  essential  communications  and 
negative  otherwise. 


DISCUSSION 

Some  relationships  were  found  between  performance  and  communication  rates  on  the 
external  circuits,  but  these  relationships  were  not  consistent  enough  or  strong  enough  to 
be  very  useful  for  assessing  team  performance.  Other  ways  of  measuring  communications 
should  be  developed.  The  simplest  and  most  economical  way  is  by  means  of  a  checklist 
that  includes  quality  and  timeliness  of  communications.  Measures  of  quality  should  cover 
correctness,  completeness,  and  relevance  of  communications  occurring  in  response  to  the 
key  events  in  an  exercise. 

There  does  not  appear  to  be  any  single  sufficient  measure  of  team  output  in  surface 
ship  ASW  exercises.  A  better  approach  is  to  use  a  checklist  that  evaluates  all  the  critical 
responses  of  team  members  and  subteams  in  an  exercise.  The  reason  for  this  is  that  a 
single  measure,  such  as  weapon  miss  distance,  reflects  only  a  small  part  of  the  many 
important  functions  that  a  team  should  perform  during  an  exercise.  Localizing  the 
submarine's  position  and  getting  the  ship  into  position  to  shoot  a  weapon  requires  the 
efforts  of  many  team  members.  A  surface  plot  of  the  situation  must  be  maintained, 
weapons  and  countermeasures  must  be  readied,  and  sonar  search  procedures  must  be 
instituted.  Prior  to  weapon  use,  approach  and  attack  tactics  must  be  worked  out  and 
implemented.  Extensive  coordination  must  be  maintained  between  subteams  and  with 
other  sirface  and  air  units.  Proper  safety  procedures  are  essential  when  entering  the 
torpedo  danger  area  or  when  firing  an  ASROC  in  the  vicinity  of  friendly  aircraft  and 
ships.  The  ship  may  lose  sonar  contact  or  the  submarine  may  employ  a  decoy,  and  a 
suitable  response  must  be  planned  and  instituted.  A  second  submarine  may  be  detected 
close  in  and  an  emergency  attack  carried  out  where  speed  is  more  important  than 
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accuracy.  Finally,  usually  late  in  the  exercise,  the  ship  may  launch  one  or  more 
deliberate  attacks  against  the  submarine  It  has  been  pursuing.  Under  these  circum¬ 
stances,  miss  distance  is  of  minor  usefulness  for  either  measurement  or  feedback. 

A  useful  checklist  for  evaluating  team  performance  should  have  a  number  of 
characteristics: 

1.  It  should  be  similar  to  the  checklist  in  previous  use  in  that  it  evaluates  specific 
actions  of  subteams  and  members.  This  would  be  particularly  useful  for  feedback  to 
teams. 

2.  It  should  have  a  simplified  tabulation  and  scoring  method  so  that  it  can  be 
completed  during  an  exercise  without  interfering  with  other  duties  of  instructors. 

3.  Unweighted  percentage  scores  should  be  used  for  feedback  to  subteams  and 
individuals.  Weighted  scores  for  qualification  grading  can  be  computed  separately  as 
appropriate. 

4.  The  checklist  should  be  based  on  a  set  of  objectives  that  are  sufficiently  precise 
to  allow  specification  of  objective  scoring  criteria. 


CONCLUSIONS 

Quantitative  measures  of  the  volume  of  communications  are  of  limited  usefulness  for 
assessing  team  and  '  .dividual  performance.  Measures  of  quality  and  timeliness  appear  to 
be  more  appropriate,  and  should  include  ways  of  assessing  incomplete,  excessive,  or  late 
communications. 

A  multiple  scoring  system  that  evaluates  team  responses  to  the  many  significant 
events  occurring  in  an  ASW  exercise  is  needed  for  team  evaluation  and  feedback.  For 
maximum  accuracy  and  objectivity,  the  system  should  employ  a  checklist  that  can  be 
completed  during  an  exercise. 


RECOMMENDATIONS 

1.  An  objective  performance  checklist  should  be  developed  for  use  in  14 A2  ASW 
training  exercises  and  for  grading  Phase  I  qualification  exercises.  Particular  attention 
should  be  given  to  assessing  the  timeliness  and  quality  of  communications.  The  checklist 
should  incorporate  the  performance  standards  given  in  the  ASW  objectives 1  developed  as 
part  of  this  project  (Slough  6c  Stern,  1981;  Stern  6c  Slough,  1981). 


‘Data  taken  from  tapes  of  communications  collected  for  this  report  were  part  of  the 
information  used  to  develop  the  objectives. 
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